{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use `on_fail` Actions\n",
    "\n",
    "`on_fail` actions are instructions to Guardrails that direct action when a validator fails. They are set at the validator level, not the guard level. \n",
    "\n",
    "The full set of on_fail actions are avialable in the [Error remediation concepts doc](https://www.guardrailsai.com/docs/concepts/error_remediation), and will not all be covered here.\n",
    "\n",
    "Instead, this interactive doc will serve to guide you on how and when to use different `on_fail` actions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/zayd/workspace/testbench/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:654: UserWarning: Can't initialize NVML\n",
      "  warnings.warn(\"Can't initialize NVML\")\n"
     ]
    }
   ],
   "source": [
    "# setup, run imports\n",
    "from guardrails import Guard, install\n",
    "\n",
    "try:\n",
    "    from guardrails.hub import DetectPII\n",
    "except ImportError:\n",
    "    install(\"hub://guardrails/detect_pii\")\n",
    "    from guardrails.hub import DetectPII"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "source": [
    "## Raise Exceptions\n",
    "\n",
    "If you do not want to change the outputs of an LLM when validation fails, you can instead wrap your guard in a try/catch block.\n",
    "\n",
    "This works particularly well for input validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/zayd/workspace/testbench/.venv/lib/python3.11/site-packages/guardrails/validator_service/__init__.py:85: UserWarning: Could not obtain an event loop. Falling back to synchronous validation.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output validation failed\n",
      "Validation failed for field with errors: The following text in your response contains PII:\n",
      "Hello, my name is John Doe and my email is john.doe@example.com\n",
      "input validation failed\n",
      "Validation failed for field with errors: The following text in your response contains PII:\n",
      "Hello, my name is John Doe and my email is john.doe@example.com\n"
     ]
    }
   ],
   "source": [
    "guard = Guard().use(DetectPII(pii_entities=\"pii\", on_fail=\"exception\"))\n",
    "\n",
    "try:\n",
    "    guard.validate(\"Hello, my name is John Doe and my email is john.doe@example.com\")\n",
    "except Exception as e:\n",
    "    print(\"output validation failed\")\n",
    "    print(e)\n",
    "\n",
    "\n",
    "# on input validation\n",
    "\n",
    "guard = Guard().use(DetectPII(pii_entities=\"pii\", on_fail=\"exception\"), on=\"messages\")\n",
    "\n",
    "\n",
    "# on input validation\n",
    "\n",
    "guard = Guard().use(DetectPII(pii_entities=\"pii\", on_fail=\"exception\"), on=\"messages\")\n",
    "\n",
    "try:\n",
    "    guard(\n",
    "        model=\"gpt-4o-mini\",\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": \"Hello, my name is John Doe and my email is john.doe@example.com\",\n",
    "            }\n",
    "        ],\n",
    "    )\n",
    "except Exception as e:\n",
    "    print(\"input validation failed\")\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `noop` to log and continue\n",
    "\n",
    "If you want to log the error and continue, you can use the `noop` action. This is useful for when you want to log the error, but not stop the LLM from running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "guarded just fine\n",
      "Check if validation passed:  False\n",
      "Show that the validated text and raw text remain the same:  True\n"
     ]
    }
   ],
   "source": [
    "# the on_fail parameter does not have to be set, as the default is \"noop\"\n",
    "\n",
    "guard = Guard().use(DetectPII(pii_entities=\"pii\", on_fail=\"noop\"))\n",
    "\n",
    "res = guard.validate(\"Hello, my name is John Doe and my email is john.doe@example.com\")\n",
    "print(\"guarded just fine\")\n",
    "print(\"Check if validation passed: \", res.validation_passed)\n",
    "print(\n",
    "    \"Show that the validated text and raw text remain the same: \",\n",
    "    res.validated_output == res.raw_llm_output,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `fix` to automatically fix the error\n",
    "\n",
    "Note, not all validators implement a 'fix' value. You can view the FailResult implementation in a validator to see if it has a fix value.\n",
    "\n",
    "Here's [an example](https://github.com/guardrails-ai/detect_pii/blob/48b15716460fe9e4e5b83ba9607ce3764d9b6d2e/validator/main.py#L188) that shows how Detect PII is written to return anonymized text as a fix value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/zayd/workspace/testbench/.venv/lib/python3.11/site-packages/guardrails/validator_service/__init__.py:85: UserWarning: Could not obtain an event loop. Falling back to synchronous validation.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Check if validated_output is valid text:  True\n",
      "Scrubbed text:  Hello, my name is <PERSON> and my email is <EMAIL_ADDRESS>\n"
     ]
    }
   ],
   "source": [
    "guard = Guard().use(DetectPII(pii_entities=\"pii\", on_fail=\"fix\"))\n",
    "\n",
    "res = guard.validate(\"Hello, my name is John Doe and my email is john.doe@example.com\")\n",
    "\n",
    "print(\"Check if validated_output is valid text: \", res.validation_passed)\n",
    "print(\"Scrubbed text: \", res.validated_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `reask` to automatically ask for an output that passes validation\n",
    "\n",
    "This reask prompt is computed from the validators themselves. It's an interpolation \n",
    "\n",
    "In order for the reask prompt to work, the following additional params must be provided:\n",
    "\n",
    "- messages\n",
    "- llm_api OR model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validated output:  Sure! Here's a fictional person without any personal identifiable information:\n",
      "\n",
      "**Name:** <PERSON>  \n",
      "**Email:** <EMAIL_ADDRESS>  \n",
      "\n",
      "Feel free to use this for any creative purposes!\n",
      "Number of reasks:  1\n"
     ]
    }
   ],
   "source": [
    "guard = Guard().use(\n",
    "    DetectPII(pii_entities=\"pii\", on_fail=\"reask\"),\n",
    ")\n",
    "\n",
    "res = guard(\n",
    "    messages=[\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"Make up a fake person and email address\",\n",
    "        }\n",
    "    ],\n",
    "    model=\"gpt-4o-mini\",\n",
    "    num_reasks=1,\n",
    ")\n",
    "\n",
    "print(\"Validated output: \", res.validated_output)\n",
    "print(\"Number of reasks: \", len(guard.history.last.iterations) - 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `custom` to anything else"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/zayd/workspace/testbench/.venv/lib/python3.11/site-packages/guardrails/validator_service/__init__.py:85: UserWarning: Could not obtain an event loop. Falling back to synchronous validation.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CUSTOM LOGIC COMPLETE!\n"
     ]
    }
   ],
   "source": [
    "# A custom on_fail can be as simple as a function\n",
    "\n",
    "\n",
    "def custom_on_fail(value, fail_result):\n",
    "    # This will turn up in validated output\n",
    "    return \"CUSTOM LOGIC COMPLETE!\"\n",
    "\n",
    "\n",
    "guard = Guard().use(\n",
    "    DetectPII(pii_entities=\"pii\", on_fail=custom_on_fail),\n",
    ")\n",
    "res = guard.validate(\"Hello, my name is John Doe and my email is john.doe@example.com\")\n",
    "\n",
    "print(res.validated_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "String validated: Hello, my name is John Doe and my email is john.doe@example.com\n",
      "\n",
      "Reason it failed: [ErrorSpan(start=18, end=26, reason='PII detected in John Doe')]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Of course, the function also has access to the fail_result and source text,\n",
    "# so interesting logic/formatting over those is also possible\n",
    "# Here, we show the specific char spans where the validator detected malfeasance\n",
    "\n",
    "\n",
    "def custom_on_fail(value, fail_result):\n",
    "    return f\"\"\"\n",
    "String validated: {value}\n",
    "\n",
    "Reasons it failed: {fail_result.error_spans}\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "guard = Guard().use(\n",
    "    DetectPII(pii_entities=\"pii\", on_fail=custom_on_fail),\n",
    ")\n",
    "res = guard.validate(\"Hello, my name is John Doe and my email is john.doe@example.com\")\n",
    "\n",
    "print(res.validated_output)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
